An object storage system (e.g. Amazon's S3) allows large volumes of data to be stored efficiently across a distributed system, at relatively low cost and with high reliability. However, when compared to file-based storage systems, object storage systems typically suffer from several limitations.
In object storage systems, objects are immutable, meaning once an object is created, the data in it cannot be changed or appended to. A new object can be associated with a given key (effectively updating the original object), but this typically requires retransmitting the entire object. However, object updates are atomic—meaning either the entire update succeeds or the entire update fails.
Object stores are typically eventually (rather than strongly) consistent. That is, if a key is associated with a new object (or is deleted), the change may not be visible to consumers for some period of time; furthermore, the particular version of an object a consumer receives for a given key is indeterminate.
Additionally in current object storage systems there is no central list of objects in the storage system (at least not one that is exposed to users), so listing, searching, or counting objects are all expensive operations.
There is a current need to overcome these limitations to create an object storage system that allows users to have increased visibility into the data being stored in the system and how the transactions are processed.